Cameras typically have a smaller dynamic range than real world scenes. Bright image parts that exceed a camera's dynamic range will appear as over-exposed or clipped areas. A camera will assign the maximum possible value of a color channel to such over-exposed or clipped areas. However, assigning this maximum value results in a loss of color, texture, contrast and luminance information in such areas.
Specular or shiny objects, such as cars or metal objects, have specular surfaces that tend to result in over-exposed or clipped image areas when imaged with a camera sensor. Specular surfaces tend to have highlights which need to be accurately (or at least plausibly) reproduced to convey the specular appearance.
Inpainting is the process of reconstructing image regions of missing image information, such as information of over-exposed or clipped regions. Inpainting typically analyzes information surrounding the missing information region and propagates or copies information from similar, surrounding regions to the missing information region. However, this is not successful when the region to be reconstructed is not similar to its surroundings. In particular, this is not successful for specular highlight regions because such regions have different properties than their surroundings.
Specular highlight regions have a peaked luminance profile which is not well-represented in neighboring image areas. Moreover, the luminance values of the over-exposed image regions exceed those of the remaining image, hence propagating or copying information from elsewhere in the image is not applicable.
The presence of over-exposed regions leads to a degraded visual appearance on conventional displays. However, this problem is exacerbated when imagery with over-exposed content is prepared for display on a high dynamic range (HDR) display. Standard dynamic range (SDR) content can generally be prepared for display on an HDR device with the aid of an inverse tone reproduction operator (also known as an inverse tone mapping operator, or iTMO for short). iTMOs expand a dynamic range non-linearly and bright image regions especially receive the largest expansion. As a consequence, featureless over-exposed regions may be emphasized and become even less visually attractive.
Pouli et al., ‘Image Statistics in Visual Computing’, A K Peters/CRC Press, 2013 provide an overview of statistical regularities found in natural images, and report that many RGB-like color spaces show strong correlations between color channels for natural images. This property can be leveraged for reconstruction of regions with one or two clipped channels. For example, Abebe et al., ‘Color Clipping and Over-exposure Correction’, Eurographics Symposium on Rendering (Experimental Ideas & Implementations track), 2015 describe a correction method, which relies on the correlation between RGB channels of color images to recover regions where one and two channels are clipped. A brightness reshaping method is used when all three channels are clipped. Abebe et al., ‘Correction of Over-Exposure using Color Channel Correlations’, IEEE GlobalSIP, 2014, describe a further method that leverages cross-channel correlations to reconstruct clipped regions.
Rempel et al., Ldr2hdr: on-the-fly reverse tone mapping of legacy video and photographs. In ACM Transactions on Graphics (TOG), August 2007, Vol. 26, No. 3, p. 39, describe an inverse tone mapping technique that includes reconstruction of over-exposed regions by applying Gaussian convolution to the mask that indicates over-exposed regions. The convolved mask is multiplied by an edge-stopping function that is applied to the same input mask, and the resulting profile is added to the inverse tone mapped image.
If no information is present, i.e. in over-exposed regions, some techniques propose to super-impose a simple Gaussian luminance profile to give shape to these regions. For example, Wang et al., High dynamic range image hallucination, In Proceedings of the 18th Eurographics conference on Rendering Techniques, June 2007 (pp. 321-326), discuss processing image luminance with a bilateral filter to produce a low frequency representation of luminance. A high frequency representation is obtained based on the residual between the image and the low frequency version. A circularly symmetric Gaussian profile is superimposed to create new luminance profiles to each clipped region. Wang requires a user to select a texture reference by marking the image with one or more strokes. The texture is propagated into the clipped area via a texture synthesis process. The new luminance profile, the texture and the original image are blended through Poisson editing to smooth transitions.
Guo et al., Correcting over-exposure in photographs, In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference, June 2010, (pp. 515-521) recreate a luminance profile through an optimization process that takes into account the likelihood of over-exposure of each pixel.
Hou et al., Recovering Over-/Underexposed Regions in Photographs, 2013, SIAM J. Imaging Sciences, 6(4), 2213-2235 discusses separating an image into lightness and chrominance information by operating in the CIE Lab color space. They propose to use inpainting lightness values by smoothing and attenuating wavelet coefficients. The lightness profile created may be compressed through tone mapping. The chromatic channels are inpainted using a similar procedure where wavelet coefficients are smoothed. An additional adjustment/normalization takes into account the lightness adjustments.
Rouf et al., Gradient domain color restoration of clipped highlights, June 2012, In Computer Vision and Pattern Recognition Workshops (CVPRW), 2012 IEEE Computer Society Conference on (pp. 7-14). IEEE, focus on hue restoration in clipped areas. For over-exposed regions, gradient values at the boundary are smoothed to avoid boundary artefacts, and to reconstruct a Gaussian profile.
Elboher et al., Recovering color and details of clipped image regions, June 2012, Proc. CGVCVIP, discuss a color correction method for clipped image regions based on color lines, i.e. linear structures identified in 3D plots of pixel values.